Eye gesture tracking

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for eye gesture recognition. In one aspect, a method includes obtaining an electrical signal that represents a measurement, by a photodetector, of an optical signal reflected from an eye and determining a depth map of the eye based on phase differences between the electrical signal generated by the photodetector and a reference signal. Further, the method includes determining gaze information that represents a gaze of the eye based on the depth map and providing output data representing the gaze information.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional of U.S. patent application Ser. No.15/359,460, entitled “EYE GESTURE TRACKING”, filed Nov. 22, 2016, whichis a continuation-in-part application of U.S. patent application Ser.No. 15/228,282, entitled “GERMANIUM-SILICON LIGHT SENSING APPARATUS,”filed Aug. 4, 2016 and issued as U.S. Pat. No. 9,954,016 on Apr. 24,2018 and claims the benefit of U.S. Provisional Application No.62/363,179 filed on Jul. 15, 2016. U.S. patent application Ser. No.15/228,282 claims the benefit of U.S. Provisional Patent Application No.62/200,652, filed Aug. 4, 2015, U.S. Provisional Patent Application No.62/209,349, filed Aug. 25, 2015, U.S. Provisional Patent Application No.62/210,946, filed Aug. 27, 2015, U.S. Provisional Patent Application No.62/210,991, filed Aug. 28, 2015, U.S. Provisional Patent Application No.62/211,004, filed Aug. 28, 2015, U.S. Provisional Patent Application No.62/217,031, filed Sep. 11, 2015, U.S. Provisional Patent Application No.62/251,691, filed Nov. 6, 2015, and U.S. Provisional Patent ApplicationNo. 62/271,386, filed Dec. 28, 2015. These prior applications areincorporated by reference in their entirety.

BACKGROUND

The present specification relates generally to eye gesture tracking.

Light can be directed towards an eye and reflected light may beobserved. The reflected light can be processed to determine informationpertaining to the eye.

SUMMARY

In some implementations, a method of eye gesture tracking can be used todetermine gaze information of an eye. The method of eye gesture trackingcan include a demodulation of modulated optical signals that arereflected from the eye. The demodulated signals can be processed togenerate a depth map of the eye and further determine the gazeinformation of the eye. The gaze information of the eye can includeinformation representing, e.g., a pupil or an iris of the eye, which, inturn, can be used for various applications such as determining userpreference data, controlling human-machine interaction apparatusesvisually, providing cross-platform peripheral control, and the like. Inaddition, by tracking the eye gesture of the eye, corresponding eye gazeinformation can be used to refocus tunable optical elements in real timeto alter light incident on the eye, creating for example nausea-freeviewing experiences. The method of eye gesture tracking can also be usedon various platforms to provide enhanced viewing experiences viadynamically refocused optical elements, providing for examplethree-dimensional (3D) foveated imaging.

One innovative aspect of the subject matter described in thisspecification is embodied in methods that include the actions ofobtaining an electrical signal that represents a measurement, by aphotodetector, of an optical signal reflected from an eye anddetermining a depth map of the eye based on phase differences betweenthe electrical signal generated by the photodetector and a referencesignal. Further, the methods can include the actions of determining agaze information that represents a gaze of the eye based on the depthmap and providing output data representing the gaze information.

Other implementations of this and other aspects include correspondingsystems, apparatus, and computer programs, configured to perform theactions of the methods, encoded on computer storage devices.

Implementations may each optionally include one or more of the followingfeatures. For instance, the methods can include providing one or morefilters to the optical signal reflected from the eye to removenon-target wavelength signals. Additionally, the methods can includeproviding one or more lenses to the optical signal reflected from theeye to focus the optical signal to the photodetector. The depth map caninclude one or more data sets of 3D information. The gaze informationcan include one or more of an identification of a particular region ofthe eye, an identification of a pupil of the eye, an identification ofan iris of the eye, or an identification of a physiological structure ofthe eye. In some aspects, providing output data representing the gazeinformation includes providing the output data representing the gazeinformation as input data to another device, machine or system.

The methods can include determining an eye gesture based on the gazeinformation and providing output data representing the eye gesture. Inthis instance, the eye gestures can include one or more of a movement ofthe eye, a rotation of the eye, a steady state of the eye, a duration ofthe steady state of the eye, a closed state of the eye, a duration ofthe closed state of the eye, an open state of the eye, a duration of theopen state of the eye, a blinking state of the eye, a duration of theblinking state of the eye, or a frequency of the blinking state of theeye. Further, providing output data representing the eye gesture caninclude providing the output data representing the gaze information asinput data to another device, machine, or system.

In certain aspects, the optical signal reflected from the eye isgenerated by one or more optical sources that are biased by a modulatedsignal, the modulated signal being in sync with the reference signal.The methods can include generating an iris vector normal to a plane thatis tangential to the eye and determining gaze information thatrepresents a gaze of the eye based on the depth map and the iris vector.The methods can also include generating a pupil position of the eye on aplane that is tangential to the eye and determining gaze information therepresents a gaze of the eye based on the depth map and the pupilposition.

Another innovative aspect of the present disclosure can be embodied in asystem including a machine with a display, the display including aplurality of tunable optical elements. The system can also include adevice including circuitry configured to obtain an electrical signalthat represents a measurement, by a photodetector, of an optical signalreflected from an eye. The circuitry can further be configured todetermine a depth map of the eye based on phase differences between areference signal and the electrical signal generated by thephotodetector, and determine a gaze information that represents a gazeof the eye based on the depth map. Additionally, the system can includeone or more processors in communication with the machine and the device,the one or more processors including one or more storage devices storinginstructions that are operable, when executed by the one or moreprocessors, to cause the one or more processors to perform theoperations including receiving, from the device, output datarepresenting the gaze information and determining the gaze informationrepresenting the gaze of the eye in relation to the display of themachine.

In some aspects, the operations can further include determining aparticular position on the display that the eye is focused on, theparticular position being based on the gaze information representing thegaze of the eye in relation to the display and providing an indicationat the particular position on the display. The operations can includedetermining a particular position on the display that the eye is focusedon, the particular position being based on the gaze informationrepresenting the gaze of the eye in relation to the display andproviding a foveated image at the particular area on the display. Theplurality of tunable optical elements can include tunable elements ortunable mirrors. In this instance, a tuning of a subset of the pluralityof tunable optical elements is activated based on the gaze informationrepresenting the gaze of the eye in relation to the display. Further,the tuning of the subset of the plurality of tunable optical elementscan include dynamically refocusing light incident on the subset of theplurality of tunable optical elements.

The system can include a wearable coupled to the machine, the device,and the one or more processors to form an integrated hardware package,the display of the machine being opaque in which visual images are shownon the display by one or more of an array of light sources. In certainaspects, the system can include a wearable coupled to the machine andthe device to form an integrated hardware package, the display of themachine being opaque in which visual images are shown on the display byone or more of an array of light sources, and the one or more processorslocated at a remote location and in communication with the integratedhardware package via a wireless or wired connection. In other aspects,the system can include a wearable coupled to the machine, the device,and the one or more processors to form an integrated hardware package,the display of the machine being at least partly transparent to imagesprojected towards the display, whereby a property of the imagesprojected towards the display is modified by one or more of theplurality of tunable optical elements of the display.

Further, the system can include a wearable coupled to the machine andthe device to form an integrated hardware package, the display of themachine being at least partly transparent to images projected towardsthe display, whereby a property of the images projected towards thedisplay is modified by one or more of the plurality of tunable opticalelements of the display, and the one or more processors located at aremote location and in communication with the integrated hardwarepackage via a wireless or wired connection. The system can also includea pluggable coupled to the device and the one or more processors to forman integrated hardware package and the machine located at a remotelocation and in communication with the integrated hardware package via awireless or wired connection, the display of the machine being opaque inwhich visual images are shown on the display by one or more of an arrayof light sources.

In some aspects, the system can include a wearable coupled to the deviceand the one or more processors to form an integrated hardware packageand the machine located at a remote location and in communication withthe integrated hardware package via a wireless or wired connection, thedisplay of the machine being opaque in which visual image are shown onthe display by one or more of an array of light sources. In thisinstance, the operations can further include determining a particularposition on the display that the eye is focused on, the particularposition being based on the gaze information representing the gaze ofthe eye in relation to the display and providing an indication at theparticular position on the display. In certain aspects, the opticalsignal reflected from the eye is generated by an optical source that isbiased by a modulated signal, the modulated signal being in sync withthe reference signal.

Another innovative aspect of the present disclosure can be embodied in adevice including a plurality of tunable optical elements for adjustingfocal lengths. The wearable device can also include one or moreprocessors including one or more storage devices storing instructionsthat are operable, when executed by the one or more processors, to causethe one or more processors to perform operations including, obtaining anelectrical signal that represents a measurement, by a photodetector, ofan optical signal reflected from an eye and determining a depth map ofthe eye based on phase differences between a reference signal and theelectrical signal generated by the photodetector. The operations canfurther include determining gaze information that represents a gaze ofthe eye based on the depth map, the gaze information representing thegaze of the eye in relation to a display of a remote device andactivating a tuning of a subset of the plurality of tunable opticalelements based on the gaze information.

Advantageous implementations can include one or more of the followingfeatures. The eye gesture tracking methods of the present disclosure canbe used to provide cross-platform peripheral control. The cross-platformperipheral control can be used to exchange information between multipledevices. The exchanged information can include eye gesture information,commands that correspond to the eye gesture information, gaze positionsof an eye, and the like. This cross-platform peripheral control can beutilized to extend the operation regions in comparison to traditionaleye tracking schemes. As such, the eye gesture tracking methods of thepresent disclosure provide greater operation regions that are notconstrained as the traditional eye tracking schemes are, due to limiteddetection regions and localization of the traditional eye trackingschemes to only a specific device. Moreover, more than one user mayapply the cross-platform peripheral control to the multiple devices atthe same time, so that a user-to-user interaction can be effectivelycreated.

Additionally, the eye gesture tracking methods of the present disclosurecan be used to provide nausea-free viewing experiences. In certainaspects, the eye gesture tracking information can be used in opticalsystems that utilize tunable optical elements to refocus imagesaccording to the eye gesture tracking information and a known distanceinformation. The tunable optical elements adjust angles of eye-incidentlight to provide real-time focusing. The real-time focusing based on theeye gesture tracking methods of the present disclosure can reducefeelings of nausea by maintaining consistent depth perception betweenthe user's eye and brain. Moreover, the eye gesture tracking informationmay be used to control a subset of the tunable optical elements creatinga foveated focusing, where focal lengths for various regions in an imagethat is presented to a viewer may be controlled to be different. Thefoveated focusing of the present disclosure provides a natural 3D effectvia simple tunable optics, unlike the traditional foveated renderingproviding an artificial 3D effect via complicated computationalalgorithms.

The details of one or more embodiments of the invention are set forth inthe accompanying drawings and the description below. Other features andadvantages of the invention will become apparent from the description,the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is an exemplary illustration of an eye gesture tracking system.

FIG. 1B is an exemplary illustration of a time-of-flight device.

FIG. 1C is an exemplary illustration of a time-of-flight device.

FIGS. 1D and 1E are example techniques for determining characteristicsof a user's eye.

FIG. 1F is an exemplary illustration of phases for charge collection.

FIG. 1G is an exemplary illustration of light emission, detection andcharge collection.

FIG. 1H is an exemplary illustration of signal voltage during chargecollection

FIG. 1I is an exemplary illustration of shifted phases for chargecollection.

FIG. 1J is an exemplary illustration of light emission, detection andphase-shifted charge collection.

FIG. 1K is an exemplary illustration of signal voltage duringphase-shifted charge collection.

FIG. 1L is an exemplary illustration of a time-of-flight device.

FIG. 2A is an exemplary illustration of a cross-platform peripheralcontrol system using eye gesture tracking.

FIG. 2B is an exemplary illustration of a cross-platform peripheralcontrol system using eye gesture tracking.

FIG. 3A is an exemplary illustration of a wearable device using eyegesture tracking.

FIG. 3B is an exemplary illustration of an optical image-refocusingsystem using a lens.

FIG. 3C is an exemplary illustration of an optical image-refocusingsystem using a mirror.

FIG. 4 is an exemplary illustration of a wearable device using eyegesture tracking.

FIG. 5A is an exemplary illustration of a stand-alone eye gesturetracking device attached to a machine.

FIG. 5B is an exemplary illustration of an embedded eye gesture trackingdevice enclosed in a machine.

FIG. 6 is a flow chart illustrating a process for eye gesture tracking.

FIG. 7 is a flow chart illustrating a process for tuning opticalelements based on eye gesture tracking.

FIG. 8 is an exemplary illustration of a computer device and a mobilecomputer device.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

Methods of eye gesture tracking can be used to determine gazeinformation pertaining to a tracked eye. The methods can includeilluminating an eye and detecting reflected optical signals from theeye, to track a gaze direction and a focus of the eye. Determination ofthe gaze direction and the focus of the eye can be useful incommunicating with another device. For example, the gaze information ofthe eye can be used to provide one or more commands to another device.In some implementations, the gaze information and/or other informationlike hand gestures can be detected by the system described hereinembedded in a cell phone, and the cell phone can be used as a remotecontrol that receives the commands from the user and connects to otherdevices such as tablet, television and etc. to execute the commands. Incertain implementations, the gaze information can include gestures ofthe eye. As such, eye gestures such as eye movement, eye rotation, eyestate, and the like, can be used to indicate certain commands to beprovided to another device. In some implementations, the gazeinformation of the eye can be used to determine the location of theeye's focus, such as where the eye is focused at a particular display.In this instance, the location of the eye's focus with respect to thedisplay can be used to gather information indicating a user's interests.For example, if an advertisement is provided at the display, the focusof the user's eye with respect to the location of the advertisementbeing provided at the display can be used to determine what the user isinterested in. As such, the location of an eye gaze, and for example thelength of time the eye holds that particular gaze, can be helpful indetermining the user's interest levels for contents being provided atthe particular display.

In some implementations of the present disclosure, the methods of eyegesture tracking can be integrated into wearables and/or peripheraldevices. For example, a wearable device can be used to provideillumination at an eye, and detect the reflected optical signals of theeye. The wearable device can include components such as anaccelerometer, a gyroscope, or both, to aid in the tracking of the eyeand the focus of the eye at a particular display so that the eyegestures can be tracked efficiently and persistently. In certainimplementations, the wearable device can further include tunable opticalelements for light path adjustments. The tunable optical elements caninclude mirrors and/or lenses that are adjusted based on the movement,or lack thereof, of the tracked eye. The tunable optical elements can beused to provide dynamic focusing and defocusing in real time to aid inthe eye's viewing of a particular object or display. For example, thetunable optical elements can be used to solve inconsistencies betweenaccommodation and vergence when viewing images at a virtual reality (VR)or augmented reality (AR) display. In certain implementations, thecomponents of the wearable device can be implemented externally in aremote device that is separate from the wearable device. The methods ofeye tracking can be used to provide data particular to the eye gaze asoutput and use this output to provide commands at remote devices and/ortunable optical elements to aid in various viewing experiences.

FIG. 1A is an exemplary illustration of an eye gesture tracking system100. The eye gesture tracking system 100 can be used to processinformation of a user's eye in response to generating a depth map of theeye. The eye gesture tracking system 100 includes an eye gesturetracking device 110 for tracking movement of a user's eye 120, agraphical display 130, a signal processing unit 140 for processing eyedata detected at the eye gesture tracking device 110, and optionally aconsole 170 providing additional user input to the system depending onthe nature of the application. The user's eye 120 can include one orboth eyes of a user that is viewing the graphical display 130.

The graphical display 130 can be one or more graphical displays on acomputer, laptop, desktop, television, smart phone, tablet and the like.The graphical display 130 can include a liquid crystal display (LCD), alight emitting diode (LED) display, an organic light emitting diode(OLED) display, a head mounted display (HMD) and the like. In someimplementations, the graphical display 130 can include tunable opticalelements such as a mirror and/or a tunable lens. In this instance, thetunable optical elements of the graphical display 130 can be configuredto adjust focusing as well as defocusing in real time to aid the user'seye 120 in viewing the graphical display 130.

The eye gesture tracking device 110 can include one or more eye gesturetracking devices in communication with the signal processing unit 140.The eye gesture tracking device 110 can provide illumination at theuser's eye 120 and receive reflected optical signals of the user's eye120. The eye gesture tracking device 110 can include a modulated opticalsource that illuminates the user's eye 120 at one or more selectedwavelengths. The modulated optical source can include a single opticalemitter or multiple optical emitters modulated by a radio-wave frequency(RF) or a microwave frequency voltage source providing the illumination.In some implementations, the optical emitters can be used to illuminatethe entirety of the user's eye 120. In other implementations, theoptical emitters can be used to illuminate selected portions of theuser's eye 120. The one or more wavelengths used in the eye gesturetracking system 100 can be predetermined based on various criteria, forexample, non-pervasiveness to the human eye, low solar irradiance at sealevel, eye safety, and the like.

In some implementations, the eye gesture tracking device 110 can includeone or more photodetectors for receiving the reflected optical signalsof the user's eye 120. The reflected optical signals of the user's eye120 can be reflections of the modulated optical signals provided by theeye gesture tracking device 110. In certain implementations, the eyegesture tracking device 110 can detect the reflected, modulated opticalsignals by the one or more photodetectors. The photodetectors may beimplemented by the techniques described in U.S. patent application Ser.No. 15/338,660 titled “High-Speed Light Sensing Apparatus,” filedOctober 31, and U.S. patent application Ser. No. 15/228,282, entitled“GERMANIUM-SILICON LIGHT SENSING APPARATUS,” filed Aug. 4, 2016.

The signal processing unit 140 can include one or more signal processingunits in communication with the graphical display 130 and the eyegesture tracking device 110. The signal processing unit 140 can beconfigured to determine gaze information 150 of the user's eye 120 viadata corresponding to the eye gesture tracking device 110 and thegraphical display 130. The eye gesture tracking device 110 can beconfigured to demodulate the reflected, modulated optical signals.Further, the eye gesture tracking device 110 can be configured to createa depth map of the illuminated portions of the user's eye 120. The depthmap can correspond to the reflected optical signals that are detected bythe photodetectors of the eye gesture tracking device 110. Specifically,the depth map can provide two-dimensional (2D) and three-dimensional(3D) information pertaining to the user's eye 120. The signal processingunit 140 can process the depth map according to data representing thetime-of-flight information of the reflected optical signals. In someimplementations, the depth map can be based on phase differences betweenthe reflected optical signals and a reference signal. For example, theeye gesture tracking device 110 can provide a comparison between thereflected optical signals and a reference signal, and can be used todetermine the depth map of the user's eye 120. The depth map can furtherinclude a 3D model representing the user's eye 120. As such, the 3D eyemodel can be generated and constructed, thereby allowing the signalprocessing unit 140 to determine the gaze information 150 of the user'seye 120.

The signal processing unit 140 can be located near the user's eye 120.For example, the signal processing unit 140 and the eye gesture trackingdevice 110 can be implemented in a single wearable device located at anearby location close to the user's eye 120. The signal processing unit140 and the eye gesture tracking device 110 can also be implemented in asingle peripheral device located at a remote location away from theuser's eye 120. In other implementations, the signal processing unit 140can be located separately from the eye gesture tracking device 110. Forinstance, the signal processing unit 140 can be located at the graphicaldisplay 130 and be in communication with the eye gesture tracking device110 implemented in a single wearable or peripheral device.

The gaze information 150 can include information such as the user's eyegaze direction and focus. The gaze information 150 can be determined bythe signal processing unit 140 with respect to the optical signalsreceived by the eye gesture tracking device 110. The gaze information150 can be used to analyze the user's eye behavior. Further, the gazeinformation 150 can be used to identify a location of the user's eye's120 focus with respect to the display 130. In this instance, the gazeinformation 150 can be used to determine particular items displayed atthe display 130 that the user's eye 120 is focused at. Thus, a user'sinterests can be determined without the need for physical actuation of aparticular device. For example, ad providers can determine the interestsof a user based exclusively on the user's eye 120, without the need foractivation/detection via a computer mouse, computer trackpad, touchscreen, or the like. In other instances, physical actuation ofparticular devices may be used to perform certain functions of the userand system interaction. Utilizing such devices may become advantageousfor the efficiency as the complexity of the interaction between thesystem and the user increases. For example, fighter jet pilots mayutilize eye gaze information 150 to identify/select targets of intereston the display 130 and use console 170 to perform tasks on the target ofinterest such as target acquisition, target priority assignment, weaponsselection, and etc.

In some implementations, the gaze information 150 can be used toindicate commands to be provided to another device. In this instance,the gaze information 150 can include eye gestures such as eye movement,eye rotation, a closed-state of the eye, an open-state of the eye, anyduration thereof, and the like. The device that receives the gazeinformation 150 may analyze the gaze information 150 in real time todetermine a command as the user's eye 120 is being dynamically trackedby the eye gesture tracking device 110.

The eye gesture tracking device 110, the graphical display 130, and thesignal processing unit 140 can be independent structures, or coupledtogether in an integrated hardware package. For example, the eye gesturetracking device 110, the graphical display 130, and the signalprocessing unit 140 can be integrated in a single hardware package inwhich the display of the graphical display 130 is opaque and visualimages are shown on the display by an array of light-emitting diodesgenerating visible light, liquid crystals filtering white light, or anyother array of light sources. In some implementations, the display ofthe graphical display 130 is at least partly transparent and visualimages are projected to the display by optical refraction, diffraction,reflection, guiding or other optical means.

In another example, the eye gesture tracking device 110 and the signalprocessing unit 140 can be integrated in a single hardware package suchas a wearable device. A wearable device may be a headset, a pair ofglasses, or any other suitable wearable device. In this instance, thewearable device communicates with a main frame or a machine in which thegraphical display 130 is embedded. Further, the main frame or themachine containing the graphical display 130 can be in communicationwith the wearable device via a wireless or wired connection.

In another example, the eye gesture tracking device 110 and the signalprocessing unit 140 can be integrated in a single hardware package suchas a pluggable device. A pluggable device may be a game box, acamcorder, or any other suitable pluggable device. In this instance, thepluggable device communicates with a main frame or a machine in whichthe graphical display 130 is embedded. Further, the main frame or themachine containing the graphical display 130 can be in communicationwith the pluggable device via a wireless or wired connection.

FIG. 1B is an exemplary illustration of a time-of-flight device. Thetime-of-flight device can be integrated into the eye gesture trackingdevice 110 and can be used to determine the depth map of the user's eye120. The time-of-flight device of FIG. 1B includes a time-of-flight(TOF) pixel 160 and two sets of transistors. As illustrated in FIG. 1B,each set of the transistors can include three switch transistors (3T),i.e., a reset transistor 162 a or 162 b, a source-follower transistor164 a or 164 b, and a selection transistor 166 a or 166 b. In some otherimplementations, other arrangements of transistors may be used toachieve similar functionalities. The TOF pixel 160 can be one or moreTOF pixels that are used to detect light. As light is detected by theTOF pixel 160, the TOF pixel determines whether charge should beprocessed by the first set of transistors or the second set oftransistors. In some aspects, a received light signal may be out ofphase with respect to an emitted light signal. In this instance, the TOFpixel can be designed to be a dual switching TOF pixel so that oneswitch is modulated in phase and the other switch is modulated 180degrees out of phase with respect to the emitted light signal toaccommodate the received, out of phase, light signal. The dual switchingTOF pixel may be implemented by the techniques described in U.S. patentapplication Ser. No. 15/338,660 titled “High-Speed Light SensingApparatus,” filed October 31, and U.S. patent application Ser. No.15/228,282, entitled “GERMANIUM-SILICON LIGHT SENSING APPARATUS,” filedAug. 4, 2016.

In certain aspects, the two sets of transistors can be fabricated withthe TOF pixel 160 on a single wafer. In this instance, the two sets oftransistors may share and occupy the same light illumination area as theTOF pixel 160 does, thereby reducing an active fill factor of the TOFdevice. The two sets of transistors may be implemented by NMOS gates.NMOS gates are utilized to reduce the size of the transistors and so theTOF device. The two sets of transistors may also be implemented by PMOSgates. PMOS gates are utilized to increase certain operation parameterssuch as providing a greater usable voltage headroom. The PMOS and NMOSimplementations of the sets of transistors will be discussed furtherherein.

FIG. 1C is an exemplary illustration of a time-of-flight device. The TOFdevice of FIG. 1C includes a first wafer and a second wafer that arebonded together via die or wafer bonding 167. The first wafer caninclude a TOF pixel 165 that is fabricated on the first wafer. The TOFpixel 165 can be used to detect light pulse information. The secondwafer can be a circuit wafer 169 that includes two sets of transistors.The circuit wafer 169 can be used to process charge as light pulseinformation is detected at the TOF pixel 165. In certainimplementations, the transistors of the circuit wafer 169 do not occupythe light illumination area, thereby increasing the active fill factorof the TOF device.

The two sets of transistors can be implemented by NMOS or PMOS gates.For example, each of the two set of transistors can be implemented byNMOS gates with a threshold voltage of 0.7 Volts. In this instance, whenthe gate voltage is supplied with 3.3 Volts, a maximum source voltage ofabout 2.6 Volts can be obtained while the NMOS gate is on. Consequently,when NMOS is used as a reset transistor, the reset voltage applied tothe TOF pixel can only be as high as 2.6 Volts that results into asmaller voltage headroom. In comparison, another example may includeeach of the two set of transistors implemented by PMOS gate with anegative threshold voltage of −0.8 Volts. In this instance, when thegate voltage is supplied with 0 Volts, a maximum source voltage of about3.3 Volts can be obtain while the PMOS gate is on. Consequently, whenPMOS is used as a reset transistor, the reset voltage applied to the TOFpixel can be as high as 3.3 Volts that results into a larger voltageheadroom.

Thus, the two sets of transistors can yield a greater usable voltageheadroom when implemented by PMOS gates. This aspect of the PMOSimplementation can be attributed in part to the negative thresholdvoltage. Further, the PMOS implementation can yield a smaller impedancewhen it turns on as a switch and passes a voltage that its value isclose to a supply voltage. As such, the PMOS implementation of the twosets of transistors provide operation benefits of the TOF device,however, the physical area of the PMOS gate is larger than that of theNMOS gate and so the PMOS implementation requires a physically largerTOF device to provide such implementation. This issue can be resolved,as shown in FIG. 1C, when the TOF pixel and the PMOS circuit areimplemented on two separate wafers, followed by a wafer or die bondingto electrically connect the two separate wafers or dies. In someimplementations, the TOF pixel as shown in FIGS. 1B and 1C may include alight absorption layer including germanium. In some implementations, theTOF pixel as shown in FIGS. 1B and 1C further includes a demodulationfunction implemented by dual switching transistors or multiple PNjunctions to achieve the demodulation function. The dual switching TOFpixel may be implemented by the techniques described in U.S. patentapplication Ser. No. 15/338,660 titled “High-Speed Light SensingApparatus,” filed October 31, and U.S. patent application Ser. No.15/228,282, entitled “GERMANIUM-SILICON LIGHT SENSING APPARATUS,” filedAug. 4, 2016.

FIG. 1D shows one example technique for determining characteristics ofthe user's eye 120. The eye gesture tracking device 110 may emit lightpulses modulated at a frequency f_(m) with a duty cycle of 50%. The eyegesture tracking device 110 may receive reflected light pulses having aphase difference Φ. A photodiode array may be controlled such that areadout circuit 1 reads the collected charge Q₁ in a phase synchronizedwith the emitted light pulses, and a readout circuit 2 reads thecollected charge Q₂ in an opposite phase with the emitted light pulses.In some implementations, the distance, D, between the eye gesturetracking device 110 and one point of the user's eye 120 may be derivedusing the equation

$\begin{matrix}{{D = {\frac{c}{4\; f_{m}}\frac{Q_{2}}{Q_{1} + Q_{2}}}},} & (1)\end{matrix}$where c is the speed of light. The eye gesture tracking device 110 mayscan the user's eye 120 to obtain a depth profile of the user's eye 120.

FIG. 1E shows another example technique for determining characteristicsof the user's eye 120. The eye gesture tracking device 110 may emitlight pulses modulated at a frequency f_(m) with a duty cycle of lessthan 50%. By reducing the duty cycle of the optical pulses by a factorof N, but increasing the intensity of the optical pulses by a factor ofN at the same time, the signal-to-noise ratio of the received reflectedlight pulses may be improved while maintaining substantially the samepower consumption for the eye gesture tracking device 110. This is madepossible when the device bandwidth is increased so that the duty cycleof the optical pulses can be decreased without distorting the pulseshape. The eye gesture tracking device 110 may receive reflected lightpulses having a phase difference Φ. The photodiode diode may becontrolled such that a readout circuit 1 reads the collected charge Q₁′in a phase synchronized with the emitted light pulses, and a readoutcircuit 2 reads the collected charge Q₂′ in a delayed phase with theemitted light pulses. In some implementations, the distance, D, betweenthe eye gesture tracking device 110 and a point of the user's eye 120may be derived using the equation

$\begin{matrix}{D = {\frac{c}{4\;{Nf}_{m}}{\frac{Q_{2}^{\prime}}{Q_{1}^{\prime} + Q_{2}^{\prime}}.}}} & (2)\end{matrix}$

FIG. 1F is an exemplary illustration of phases for charge collection.The phases for charge collection represent phases in which light pulsesare emitted and electrical charge are collected by the eye gesturetracking device 110. The phases for charge collection include a 0 degreephase, a 90 degree phase, a 180 degree phase, and a 270 degree phase,and a controllable phase shift φ. The phase difference Φ may be observedbetween light pulses emitted by the eye gesture tracking device 110 andlight pulses received by the eye gesture tracking device 110. In someimplementations, the phase difference Φ occurs due to a distance betweenthe user's eye 120 and the eye gesture reading device 110. A small phasedifference can make it difficult for the eye gesture tracking device 110to efficiently detect a gesture recognition of the user's eye 120, amapping of the user's eye 120, and the like. As such, it can bebeneficial to add a phase shift φ to the collected charge so that theeye gesture recognition can be performed efficiently.

FIG. 1G is an exemplary illustration of light detection and chargecollection. The light detection and charge collection includes timesteps of light emission, light detection, and charge collection at theeye gesture reading device 110. At each of the time steps, data arecollected to represent the received light, the charge collected at the 0degree phase, the charge collected at the 90 degree phase, the chargecollected at the 180 degree phase, and the charge collected at the 270degree phase. The collection of charge at each phase can indicate anamount of collected charge at each of the received phases. In thisinstance, the amount of collected charge at each time step of each phasecan impact an accuracy of the eye gesture reading device 110 in mappingthe user's eye 120.

For example, the eye gesture tracking device 110 may emit light pulsesmodulated at a frequency f_(m) with a duty cycle of 50%. The eye gesturetracking device 110 may receive reflected light pulses having a phasedifference Φ. The TOF pixels can be controlled such that a first readoutcircuit of the eye gesture tracking device 110 reads the collectedcharge, Q0, at a phase that is synchronization with the emitted lightpulses, thus corresponding to the 0 degree phase. The eye gesturetracking device 110 can also include a second readout circuit that readsthe collected charge, Q180, at an opposite phase of the emitted lightpulses, such as the 180 degree phase. In another time step, the TOFpixels are controlled such that first readout circuit reads thecollected charge, Q90, in a quadrature phase with respect to the emittedlight pulses, such as the 90 degree phase. In this instance, the secondreadout circuit can read the collected charge, Q270, in the otherquadrature phase with respect to the emitted light pulses, such as the270 degree phase. In some implementations, the distance between the eyegesture tracking device 110 and the user's eye 120 may be derived usingthe following two equations:

$\begin{matrix}{{D = {\frac{c}{8\; f_{m}}( {1 + \frac{{Q\; 180} - {Q\; 0}}{{{{Q\; 0} - {Q\; 180}}} + {{{Q\; 90} - {Q\; 270}}}}} )}},{or}} & (3) \\{D = {\frac{c}{8\; f_{m}}{( {3 + \frac{{Q\; 0} - {Q\; 180}}{{{{Q\; 0} - {Q\; 180}}} + {{{Q\; 90} - {Q\; 270}}}}} ).}}} & (4)\end{matrix}$

Referring again to FIG. 1G, in a condition of the small phase differenceΦ between light pulses emitted by the eye gesture tracking device 110and light pulses received by the eye gesture tracking device 110, thecharge collection at the 0 degree phase is the greatest over theprovided time steps, and the charge collection at the 180 degree phaseis the lowest over the provided time steps. Such a large difference incharge collection can impact the accuracy of the charge collection as awhole. Thus, introducing phase shift φ can be helpful in eye gesturedetection by reducing the differences in charge collection at each phaseto enable a more accurate depth map of the user's eye 120.

FIG. 1H is an exemplary illustration of signal voltage during chargecollection. The signal voltage during charge collection illustrates thechange in signal voltage of multiple phases over time. Specifically,FIG. 1H illustrates the change in signal voltage for the 0 degree phase,the 90 degree phase, the 180 degree phase, and the 270 degree phase. Thedecrease in signal voltage of each phase over time represents an amountof charge that is stored for a particular phase over an interval oftime. As shown in FIG. 1H, the signal voltage of the 180 degree phase ismuch higher than the signal voltage of the 0 degree phase. Thus, the 180degree phase includes a lower rate of charge storage than that of the 0degree phase. In this instance, the accuracy of detection of the user'seye 120 by the eye gesture tracking device 110 can be negativelyimpacted due to the differences between the rates of charge storageacross the different phases. As such, it may be beneficial to include aphase shift φ in the received light signals to aid in the chargecollection so that a more accurate depth map of the user's eye 120 maybe performed.

FIG. 1I is an exemplary illustration of shifted phases for chargecollection. The shifted phases for charge collection include a 45 degreephase, a 135 degree phase, a 225 degree phase, and a 315 degree phase.The phase difference Φ may be observed between light pulses emitted bythe eye gesture tracking device 110 and light pulses received by the eyegesture tracking device 110. In some implementations, the phasedifference Φ occurs due to a distance between the user's eye 120 and theeye gesture reading device 110. A small phase difference can make itdifficult for the eye gesture tracking device 110 to efficiently detecta gesture recognition of the user's eye 120, a mapping of the user's eye120, and the like. As such, a phase shift φ of 45 degree is illustratedin FIG. 1I to the collected charge so that all phases may be offset bythe same phase shift φ of 45 degree.

FIG. 1J is an exemplary illustration of light detection andphase-shifted charge collection. The light detection and phase-shiftedcharge collection includes time steps of light emission, lightdetection, and charge collection at the eye gesture reading device 110.At each of the time steps, data is collected to represent the receivedlight, the charge collected at the 45 degree phase, the charge collectedat the 135 degree phase, the charge collected at the 225 degree phase,and the charge collected at the 335 degree phase. The collection ofcharge at each phase can indicate an amount of collected charge at eachof the received phases. In this instance, the amount of collected chargeat each time step of each phase can impact an accuracy of the eyegesture reading device 110 in mapping the user's eye 120.

For example, the eye gesture tracking device 110 may emit light pulsesmodulated at a frequency f_(m) with a duty cycle of 50%. The eye gesturetracking device 110 may receive reflected light pulses having a phasedifference Φ. The TOF pixels can be controlled such that a first readoutcircuit of the eye gesture tracking device 110 reads the collectedcharge, Q45, at a shifted-phase with respect to the emitted lightpulses, such as the 45 degree phase. The eye gesture tracking device 110can also include a second readout circuit that reads the collectedcharge, Q225, at a shifted-phase with respect to the emitted lightpulses, such as the 225 degree phase. In another time step, the TOFpixels are controlled such that first readout circuit reads thecollected charge, Q135, in the phase shift of 135 degrees with respectto the emitted light pulses. In this instance, the second readoutcircuit can read the collected charge, Q315, in the phase shift of 315degrees with respect to the emitted light pulses. In someimplementations, the distance between the eye gesture reading device 110and the user's eye 120 may be derived using the following two equations:

$\begin{matrix}{{D = {\frac{c}{8\; f_{m}}( {\frac{3}{2} + \frac{{Q\; 225} - {Q\; 45}}{{{{Q\; 45} - {Q\; 225}}} + {{{Q\; 135} - {Q\; 315}}}}} )}},{or}} & (5) \\{D = {\frac{c}{8\; f_{m}}{( {\frac{7}{2} + \frac{{Q\; 45} - {Q\; 225}}{{{{Q\; 45} - {Q\; 225}}} + {{{Q\; 135} - {Q\; 315}}}}} ).}}} & (6)\end{matrix}$

Referring again to FIG. 1J, in a condition of small phase difference Φbetween light pulses emitted by the eye gesture tracking device 110 andlight pulses received by the eye gesture tracking device 110, the chargecollected at the 45 degree phase and at the 225 degree phase are closerover the provided time steps. In comparison to FIG. 1G, in which thecharge collections are not performed with a phase shift φ and the chargecollected at the 0 degree phase and at the 180 degree phase are quitedifferent, the phase-shifted charge collection of FIG. 1J providesgreater eye mapping performance due to a lower difference in chargecollection at each phase in comparison. As differences in chargecollection can impact the accuracy of the charge collection as a whole,phase shifts can be helpful in eye gesture detection by reducing thedifference in charge collection at each phase to enable a more accuratedepth map of the user's eye 120.

FIG. 1K is an exemplary illustration of signal voltage duringphase-shifted charge collection. The signal voltage during phase-shiftedcharge collection illustrates the change in signal voltage of multiplephases over time. Specifically, FIG. 1K illustrates the change in signalvoltage for the 45 degree shifted-phase, the 135 degree shifted-phase,the 225 degree shifted-phase, and the 315 degree shifted-phase. Thedecrease in signal voltage of each phase over time represents an amountof charge that is stored for a particular phase over an interval oftime. As shown in FIG. 1K, the signal voltage of the shifted phasesincludes a more similar average rate of the signal voltage drop,compared to a more different average rate of the signal voltage dropshown in FIG. 1H. The similarity in drop rates of the signal voltage ofthe shifted phases can enable a greater accuracy of eye gesturedetection and mapping of the user's eye. As such, it may be beneficialto include a phase shift φ to the charge collection to aid in the chargecollection so that a more accurate reading of the user's eye 120 may beperformed.

FIG. 1L is an exemplary illustration of a TOF device. The TOF deviceincludes a TOF pixel 190, two capacitors 192 a and 192 b, and two setsof transistors 194 and 196. Each set of the transistors can include fiveswitch transistors (5T). In some other implementations, otherarrangements of transistors may be used to achieve similarfunctionalities. The TOF pixel 190 can be one or more TOF pixels thatare used to detect light. The charge generated by the TOF pixel 190 canbe collected by the two capacitors 192 a and 192 b. Transistors M1˜M4,which may be implemented by NMOS, PMOS, or any combination of NMOS andPMOS, are used to redistribute the collected charge by resetting thecommon-mode charge and connect the common-mode voltage to VREF. Thevoltage VREF may be the operation voltage of the TOF device 190 or apredetermined voltage depending on design constraints. Transistors M5and M6, which may be implemented by NMOS, PMOS, or any combination ofNMOS and PMOS, are used to reset the collected charge and connect themto VREF2. The voltage VREF2 may be the same voltage as VREF, theoperation voltage of the TOF device 190, or a predetermined voltagedepending on design constraints.

FIG. 2A is an exemplary illustration of a cross-platform peripheralcontrol system using eye gesture tracking. The cross-platform peripheralcontrol system using eye gesture tracking can include a wearable devicesuch as a headset 201, and a connected device such as a phone 220, atablet 230, a computing device 240, and/or a television 250, incommunication with the headset 201. The headset 201 can be used by apair of user's eyes 216A and/or 216B for viewing a connected device suchas the phone 220, the tablet 230, the computing device 240, and/or thetelevision 250. The headset 201 can include an eye tracking gesturedevice and a signal processing unit implemented in an eye-trackingmodule 213 for tracking gestures of one of the user's first and secondeyes 216A and 216B, an accelerometer 211 and gyroscope 212 fordetermining a head position of the user, a wireless communication unit214 for communicating with a connected device such as the phone 220and/or the tablet 230 and/or the computing device 240 and/or thetelevision 250, and a transparent lens 218. In some implementations, thetransparent lens 218 may include one or more tunable elements foradjustment based on the tracking of the user's eyes 216A and/or 216B.

Here, the eye-tracking module 213 can be used to illuminate the user'seye 216A with optical signals, and detect optical signals that arereflected from the user's eye 216A. The detected optical signals can beused to determine gaze information pertaining to the user's eye 216A.The gaze information can include the user's gaze with respect to thedisplays of the connected device. The gaze information can also includecommands corresponding to gestures of the user's eye 216A. The eyegesture commands can be provided as input commands to the connecteddevice. In some implementations, the eye-tracking module 213 can be usedto illuminate both of the user's eyes 216A and 216B with opticalsignals, and detect optical signals that are reflected from the user'seyes 216A and 216B to determine gaze information of both eyes 216A and216B.

The accelerometer 211 and gyroscope 212 can be used to detect anorientation of the user's head. The orientation of the user's head canbe used in effectively determining the gaze information. Further, theaccelerometer 211 and the gyroscope 212 can be used to track movementsof the user's head. Thus, any potential head movements of the user canbe identified so that the gaze information is not misrepresentedaccording to movements of the user's head.

The wireless communication unit 214 can be used to establish aconnection between the headset 201, the phone 220, the tablet 230, thecomputing device 244, and/or the television 254 via a network. Thenetwork can include Wi-Fi, BLUETOOTH, BLUETOOTH LOW ENERGY (BLE), alocal area network (LAN), and the like.

The transparent lens 218 can be used to aid the user's eyes 216A and216B in viewing the displays of the phone 220, the tablet 230, thecomputing device 240, and/or the television 250. The transparent lens218 can include tunable optical elements that can be tuned based on thedetermined gaze information representing the tracking of the user's eyes216A and 216B. In some implementations, the entirety of the transparentlens 218 can be tuned based on the gaze information. In otherimplementations, selected portions of the transparent lens 218 can betuned based on the gaze information. For example, the selected portionsof the tunable optical elements of the transparent lens 218 can be tunedto provide foveated images of particular locations at the display of thephone 220, the particular locations at the phone 220 being based onlocations at the display that the gaze of the user's eyes 216A and 216Bare directed to.

In some implementations, the phone 220 can include an accelerometer 221and gyroscope 222 for determining the orientation of the phone 220, aswell as a wireless communication unit 224 for communicating with theheadset 201. The accelerometer 221 and the gyroscope 222 of the phone220 can aid in tracking the location as well as the movement of thephone 220. By tracking the location and movement of the phone 220, theheadset 201 can effectively determine the gaze information of the user'seyes 216A and 216B when comparing the user's focus to the location ofthe phone 220. The location and the movement of the phone 220 can betransmitted from the phone 220 to the headset 201 via the wirelesscommunication device 224.

In some implementations, the tablet 230 can include an accelerometer 231and a gyroscope 232 for determining the orientation of the tablet 230,and a wireless communication unit 234 for communicating with the headset201. The accelerometer 231 and the gyroscope 232 of the tablet 230 canaid in tracking the location and the movement of the tablet 230. Intracking the location and movement of the tablet 230, the headset 201can effectively determine a reference gaze point 236 of the user's eye216A. The location and the movement of the tablet 230 can be transmittedfrom the tablet 230 to the headset 201 via the wireless communicationunit 234.

The computing device 240 can include a wireless communication unit 244for communicating with the headset 201. Additionally, the television 250can include a wireless communication unit 254 for communication with theheadset 201.

FIG. 2B is an exemplary illustration of a cross-platform peripheralcontrol system using eye gesture tracking. The cross-platform peripheralcontrol system can include a wearable device such as a headset 202, anda connected device such as a phone 220, a tablet 230, a computing device240, and/or a television 250, in communication with the headset 202. Theheadset 202 can be used by a user's eyes 216A and 216B for viewing theconnected device. The headset 202 can include two eye tracking gesturedevice and signal processing unit pairs, the first pair implemented in afirst eye-tracking module 213A and the second pair implemented in asecond eye-tracking module 213B, for tracking gestures of the user'seyes 216A and 216B, an accelerometer 211 and gyroscope 212 fordetermining a head position of the user, a wireless communication unit214 for communicating with the connected device, a first transparentlens 218A including one or more tunable elements, and a secondtransparent lens 218B including one or more tunable elements.

The first eye-tracking module 213A can be used to illuminate the firstuser's eye 216A with optical signals, and detect optical signals thatare reflected from the first user's eye 216A. The detected opticalsignals can be used to determine gaze information pertaining to thefirst user's eye 216A. The gaze information can include the gaze of theuser's first eye with respect to the displays of connected device suchas 220, 230, 240 and 250. The gaze information can also include commandscorresponding to gestures of the first user's eye 216A. The eye gesturecommands can be provided as input commands to the connected device.

The second eye-tracking module 213B can be used to illuminate the seconduser's eye 216B and detect optical signals that are reflected from thesecond user's eye 216B. The detected optical signals can be used todetermine gaze information pertaining to the second user's eye 216B. Thegaze information can include the gaze of the second user's eye 216B withrespect to the displays of the connected device. The gaze informationcan also include commands corresponding to gestures of the second user'seye 216B. The eye gesture commands can be provided as input commands tothe connected device.

The first transparent lens 218A can be used to aid the first user's eye216A in viewing the displays of the connected device. The firsttransparent lens 218A can include tunable elements that can be tunedbased on the determined gaze information representing the tracking ofthe first user's eye 216A. In some implementations, the entirety of thefirst transparent lens 218A can be tuned based on the gaze information.In other implementations, selected portions of the first transparentlens 218A can be tuned based on the gaze information. For example, theselected portions of tunable optical elements of the first transparentlens 218A can be tuned to foveated images of particular locations at thedisplay of the computing device 240, the particular locations at thecomputing device 240 being based on locations at the display that thefirst user's eye 216A is focused on.

The second transparent lens 218B can be used to aid the second user'seye 216B in viewing the connected device. The second transparent lens218B can include tunable optical elements that are tuned based on thedetermined gaze information representing the tracking of the seconduser's eye 216B. In some implementations, the entirety of the secondtransparent lens 218B can be tuned based on the gaze information. Inother implementations, selected portions of the second transparent lens218B can be tuned based on the gaze information. For example, theselected portions of tunable elements of the second transparent lens218B can be tuned to provide enhanced focus of viewing of particularlocations at the display of the computing device 240, the particularlocations at the computing device 240 being based on locations that thesecond user's eye 216B is focused on.

In certain implementations, the first user's eye 216A and the seconduser's eye 216B can be focused at a single location. For example, theuser's eyes 216A and 216B can include a reference gaze 246 located atthe display of the computing device 240 such as a laptop or desktop.Although the reference gaze 246 may be directed towards a single pointat the display of the laptop or desktop, the tunable optical elements ofeach of the first transparent lens 218A and the second transparent lens218B can be tuned independently based on the determined gaze informationof the first and second user's eyes 216A and 216B.

FIG. 3A is an exemplary illustration of a wearable device 300 using eyegesture tracking. The wearable device 300 can include a mono-visionwearable device that provides light path adjustments based on eyegesture tracking. The wearable device 300 includes a transparent or anopaque screen 310 that a user can look through or look at, tunableoptical elements 330 for adjusting a light path at the transparent orthe opaque screen 310, a wireless communication unit 340 forcommunicating with remote devices, an image projector 350 for projecting2D visuals through or at the transparent or the opaque screen 310, andan eye gesture tracking module 360 for tracking eye gestures of theuser's eyes 320A and 320B and determining depth maps corresponding toeach of the user's eyes 320A and 320B.

The gaze 325 of the user's eyes 320A and 320B can be determined by theeye gesture tracking module 360. In certain implementations, onlycertain portions of the transparent or the opaque screen 310 are tunedaccording to the gaze 325 of the user's eyes 320A and 320B. The gazeinformation corresponding to the gaze 325 can be used by the eye gesturetracking module 360 to tune a selected portion of the transparent or theopaque screen 310 such as the multiple tunable optical elements 330. Thetunable optical elements 330 can be tuned to adjust thefocusing/defocusing of a particular light path passing through theparticular portion of the transparent or the opaque screen 310. Thetunable optical elements 330 can include tunable mirrors, tunablelenses, tunable gratings or any other suitable tunable optical elementsand any combination thereof. The tunable optical elements 330 can beadjusted based on the gaze information corresponding to the gaze 325 sothat real time focusing/defocusing can be provided at the wearabledevice 300.

The real-time focusing/defocusing of the tunable optical elements 330can be used to solve inconsistencies between accommodation and vergencewhen viewing displays. For example, traditional VR experiences can causefeelings of nausea due to inconsistent depth perception mechanisms. Oneinconsistent depth perception mechanism arises when the focus of auser's eyes (accommodation) feels that images are at the same distanceof a display, while the images are simultaneously felt at differentdepths by the convergence of the user's eyes (vergence). Theseconflicting feelings that are perceived by the user can cause feelingsof sickness due to the inconsistency between accommodation and vergence.

To solve this inconsistent depth perception mechanism, among otherinconsistencies, the present method of eye tracking can be implementedin a wearable device such as the wearable device 300. The wearabledevice 300 can refocus light based on eye gaze information to adjust theangle of eye-incident light passing through or at selected portions ofthe transparent or the opaque screen 310. Thus, the tunable opticalelements 330 of the transparent or the opaque screen 310 can beconfigured to refocus light based on the determined gaze information ofthe user's eyes 320A and 320B, thereby providing a solution to theinconsistencies that can arise between accommodation and vergence duringcertain viewing experiences.

FIG. 3B is an exemplary illustration of an optical image-refocusingsystem using a lens. The optical image-refocusing system using a lensillustrates the use of a lens to refocus an object illusion according togaze information of a user's eye.

At instance 1 of the optical image-refocusing system using a lens, theuser's eye 320 is viewing the object 370 through a medium such as airwithout using a screen such as a VR display. The user's eye 320 may notbe viewing the object 370 through a transparent lens. Further, theuser's eye 320 is viewing the real object 370, rather than a virtualrepresentation of the object.

At instance 2 of the optical image-refocusing system using a lens, theuser's eye 320 is viewing the object illusion 375 through the screen380. In this instance, an image projector may be projecting a virtualrepresentation of the object 370 as the object illusion 375 through thescreen 380. In this instance, the user's eye 320 may be experiencing aninconsistency between accommodation and vergence.

At instance 3 of the optical image-refocusing system using a lens, theuser's eye 320 is viewing the object illusion 375 through a lens 330situated between the user's eye 320 and the screen 380. The lens 330 canbe a fixed lens that is used to refocus the object illusion 375. Inother implementations, the lens 330 can be a tunable lens that is usedto dynamically refocus the object illusion 375 through the screen 380 inreal time. In this instance, the lens 330 can be tuned based ondetermined gaze information of the user's eye 320.

FIG. 3C is an exemplary illustration of an optical image-refocusingsystem using a mirror. The optical image-refocusing system using amirror illustrates the use of a mirror to refocus an object illusionaccording to gaze information of a user's eye.

At instance 1 of the optical image-refocusing system using a mirror, theuser's eye 320 is viewing the object 370 through a medium such as airwithout using a screen such as a VR display. The user's eye 320 may notbe viewing the object 370 through a transparent lens. Further, theuser's eye 320 is viewing the real object 370, rather than a virtualrepresentation of the object.

At instance 2 of the optical image-refocusing system using a mirror, theuser's eye 320 is viewing the object illusion 376 through the screen380. In this instance, an image projector may be projecting a virtualrepresentation of the object 370 as the object illusion 376 through thescreen 380. In this instance, the user's eye 320 may be experiencing aninconsistency between accommodation and vergence.

At instance 4 of the optical image-refocusing system using a mirror, theuser's eye 320 is viewing the object illusion 376 through a screen 380that includes a mirror 385. The mirror 385 can be a fixed mirror that isused to refocus the object illusion 376. In other implementations, themirror 385 can be a tunable mirror that is used to dynamically refocusthe object illusion 376 through the screen 380 that includes the mirror385 in real time. In this instance, the mirror 385 can be tuned based ondetermined gaze information of the user's eye 320.

FIG. 4 is an exemplary illustration of a wearable device 400 using eyegesture tracking. The wearable device 400 using eye gesture tracking caninclude a stereo-vision wearable device that provides light pathadjustments based on eye gesture tracking. The wearable device 400includes a first transparent or opaque screen 410A and a secondtransparent or opaque screen 410B that the user can look through or lookat, a first set of tunable optical elements 430A located for adjusting alight path at the first transparent or opaque screen 410A and a secondset of tunable optical elements 430B for adjusting a light path at thesecond transparent or opaque screen 410B. The wearable device 400 mayfurther include a first wireless communication unit 440A forcommunicating with remote devices or a second wireless communicationunit 440B for communication with the remote devices, a first imageprojector 450A for projecting 2D visuals through or at the firsttransparent or opaque screen 410A, a second image projector 450B forprojecting 2D visuals through or at the second transparent or opaquescreen 410B, a first eye gesture tracking module 460A for tracking eyegestures of the first user's eye 420A and determining a depth mapcorresponding to the first user's eye 420A, and a second eye gesturetracking module 460B for tracking eye gestures of the second user's eye420B and determining a depth map corresponding to the second user's eye420B.

The wearable device 400 may further include one continuous or twoseparate transparent or opaque screens 410A and 410B which enable twodistinct gaze points 425A and 425B to be determined. As each of theuser's eyes 420A and 420B is tracked separately by each respective eyegesture tracking module 460A and 460B, the first and second opticalelements 430A and 430B can be tuned independent from one another.Further, each of the image projectors 450A and 450B can operateindependently. As such, a portion can be selected at each of thetransparent or opaque screens 410A and 410B to refocus light incident oneach of the user's eyes 420A and 420B. In this instance, 3D projectionscan be interpreted by the user's eyes 420A and 420B via the simultaneousprojection of multiple images through or at both of the transparent oropaque screens 410A and 410B.

FIG. 5A is an exemplary illustration of a stand-alone eye gesturetracking device attached to a machine. The stand-alone eye gesturetracking device is implemented as a stand-alone peripheral device 530located in proximity of a machine such as a display device 520. Thestand-alone eye gesture tracking device attached to a machine includes adisplay device 520 in communication with a stand-alone peripheral device530 that is located at a remote location away from a user's eyes 510Aand 510B.

The stand-alone peripheral device 530 includes a mechanical module 532to control the direction of light emission and detection from an eyegesture tracking module 534, so that the user's eyes are always locatedby the stand-alone peripheral device 530. The eye gesture trackingmodule 534 tracks the eye gestures of the user's eyes 510A and 510B anddetermines gaze information corresponding to the user's eyes 510A and510B. The display device 520 can include a gaze reference point 515corresponding to a focus of the user's eyes 510A and 510B with respectto the display device 520. The gaze reference point 515 can bedetermined by the eye gesture tracking module 534 of the stand-aloneperipheral device 530. In certain implementations, the display device520 can include tunable optical elements that are tuned based on thegaze reference point 515. The tunable optical elements can includetunable mirrors located at the display device 520. In otherimplementations, the display device 520 can include fixed opticalelements such as fixed mirrors for light path refocusing.

The eye gesture tracking module 530 can be configured to provide outputdata to the display device 520. The output data can include gazeinformation of the user's eyes 510A and 510B. The gaze information canbe used by the display device 520 to render an image at a particularportion of the display corresponding to the gaze reference point 515 ofthe user. The rendered image can be shown at the display of the displaydevice 520 by an array of light-emitting diodes generating visiblelight, liquid crystals filtering white light, or any other array oflight sources located at the display of the display device 520. Further,the rendered image can be shown at the display of the display device 520by optical refraction, diffraction, reflection, guiding, or any otheroptical techniques.

FIG. 5B is an exemplary illustration of an eye gesture tracking deviceembedded in a machine. The eye gesture tracking device embedded in amachine includes an embedded peripheral device 545 integrated into amachine such as a display device 540. The embedded peripheral device 545can include a mechanical module 546 to control the direction of lightemission and detection from an eye gesture tracking module 547, so thatthe user's eyes are always located by the embedded peripheral device545. The eye gesture tracking module 547 tracks the eye gestures of theuser's eyes 510A and 510B and determines gaze information correspondingto the user's eyes 510A and 510B.

The display device 540 can further include a gaze reference point 555representing the location at the display device 540 in which the user'seyes 510A and 510B are focused at. In certain implementations, thedisplay device 540 can include tunable optical elements that are tunedbased on the gaze reference point 555. The tunable optical elements caninclude tunable mirrors located at the display device 540. In otherimplementations, the display device 540 can include fixed opticalelements such as fixed mirrors.

In some implementations, the distance between the eye 510A or 510B andthe eye tracking module 534 and 547 can be determined based on TOFconcept or by other methods such as imaging processing ortri-angulation. The optical emission power can be adjusted accordinglybased on the distance between the eye 510A or 510B and the eye trackingmodule 534 and 547. For example, the optical emission power can bedynamically lowered to reduce the exposure of the eye 510A or 510B tothe optical emission given a close distance between the eye 510A or 510Band the eye tracking module 534 and 547.

FIG. 6 is a flow chart illustrating a process 600 for eye gesturetracking, according to certain exemplary implementations. The process600 for eye gesture tracking describes a process of monitoring themovement of an eye based on a generated depth map of the eye. At step610, an electrical signal is obtained that represents a measurement ofan optical signal reflected from an eye. The optical signal can beprovided by an optical source. The optical source can be biased by amodulated voltage signal that is in sync with a predetermined referencesignal. As such, the optical source can provide an optical signal in thedirection of the eye, to be reflected back from the eye.

The reflected optical signal can be received by one or morephotodetectors. In some implementations, the received optical signal canbe filtered to remove certain wavelengths. For example, one or morefilters can be provided to filter the optical signal so that only targetwavelengths remain in the filtered optical signal. In certainimplementations, one or more lenses can be provided to focus the opticalsignal before it is received by the photodetector. The lenses can betransparent lenses, fixed lenses, tunable lenses, lenses based onphotonic gratings, and the like.

At step 620, a depth map is determined based on phase differencesbetween the received optical signal and the reference signal. Thereceived optical signal can be compared to the reference signal as it isreceived. In other implementations, the received optical signal can befiltered and then a comparison can be provided between the filteredoptical signal and the reference signal. The depth map can include oneor more data sets of 3D information corresponding to the eye. In someimplementations, a 3D representation of the eye can be generatedaccording to the 3D information of the depth map. The depth map can bedetermined persistently in real time. The depth map can also bedetermined and updated at predetermined points in time. For example, thedepth map can be determined and updated every micro-second, millisecond,every second, or the like.

At step 630, gaze information is determined based on the depth map. Thegaze information can represent a gaze of the eye based on the depth map.In some implementations, the gaze information can be determined based onthe provided comparison between the reference signal and the reflectedoptical signal. Further, the gaze information can include one or more ofan identification of a particular region of the eye, an identificationof a pupil of the eye, an identification of an iris of the eye, or anidentification of a physiological structure of the eye. In certainimplementations, eye gestures of the eye can be determined from the gazeinformation. The eye gesture information can include one or more of amovement of the eye, a rotation of the eye, a steady state of the eye, aduration of the steady state of the eye, a closed state of the eye, aduration of the closed state of the eye, an open state of the eye, aduration of the open state of the eye, a blinking state of the eye, aduration of the blinking state of the eye, or a frequency of theblinking state of the eye.

The depth map can be used to generate an iris vector normal to a planethat is tangential to the eye. In this instance, the gaze informationcan be determined based on the iris vector and the depth map. The depthmap can also be used to generate a pupil position of the eye on a planethat is tangential to the eye. In this instance, the gaze informationcan be determined based on the pupil position of the eye and the depthmap.

At step 640, the gaze information of the eye is provided as output data.The output data representing the gaze information can be transmitted toa device, a machine, a system, and the like. In this instance, the gazeinformation can be transmitted as input data to the device, machine, orsystem. Additionally, eye gestures determined from the gaze informationcan be provided as output data. The eye gestures can be used to providecommands to or interact with devices, machines, or systems. For example,if the eye is tracked to have blinked three times in quick succession,this may indicate a command to be provided to a remote device such as atelevision. As such, the television may be configured to turn off if theeye is tracked to blink several times in rapid succession.

FIG. 7 is a flow chart illustrating a process 700 for tuning opticalelements based on eye gesture tracking, according to certain exemplaryimplementations. The process 700 for tuning optical elements based oneye gesture tracking describes a process of monitoring the movement ofan eye and tuning optical elements based on the eye movement. At step710, an electrical signal is obtained that represents a measurement ofan optical signal reflected from an eye. The optical signal can beprovided by an optical source. The optical source can be biased by amodulated voltage signal that is in sync with a predetermined referencesignal. As such, the optical source can provide an optical signal in thedirection of the eye, to be reflected back from the eye.

The reflected optical signal can be received by one or morephotodetectors. In some implementations, the received optical signal canbe filtered to remove certain wavelengths. For example, one or morefilters can be provided to filter the optical signal so that only targetwavelengths remain in the filtered optical signal. In certainimplementations, one or more lenses can be provided to focus the opticalsignal before it is received by the photodetector. The lenses can betransparent lenses, fixed lenses, tunable lenses, lenses based onphotonic grating, and the like.

At step 720, a depth map is determined based on phase differencesbetween the received optical signal and the reference signal. Thereceived optical signal can be compared to the reference signal as it isreceived. In other implementations, the received optical signal can befiltered and then a comparison can be provided between the filteredoptical signal and the reference signal. The depth map can include oneor more data sets of 3D information corresponding to the eye. In someimplementations, a 3D representation of the eye can be generatedaccording to the 3D information of the depth map. The depth map can bedetermined persistently in real time. The depth map can also bedetermined and updated at predetermined points in time. For example, thedepth map can be determined and updated every micro-second, millisecond,every second, or the like.

At step 730, gaze information is determined based on the depth map. Thegaze information can represent a gaze of the eye based on the depth map.The gaze information can include one or more of an identification of aparticular region of the eye, an identification of a pupil of the eye,an identification of an iris of the eye, or an identification of aphysiological structure of the eye. In certain implementations, eyegestures of the eye can be determined from the gaze information. The eyegesture information can include one or more of a movement of the eye, arotation of the eye, a steady state of the eye, a duration of the steadystate of the eye, a closed state of the eye, a duration of the closedstate of the eye, an open state of the eye, a duration of the open stateof the eye, a blinking state of the eye, a duration of the blinkingstate of the eye, or a frequency of the blinking state of the eye.

The depth map can further be used to generate an iris vector normal to aplane that is tangential to the eye. In this instance, the gazeinformation can be determined based on the iris vector and the depthmap. The depth map can also be used to generate a pupil position of theeye on a plane that is tangential to the eye. In this instance, the gazeinformation can be determined based on the pupil position of the eye andthe depth map.

At step 740, tuning of tunable optical elements is activated based onthe determined gaze information. The gaze information can be used totune particular tunable optical elements. For example, the gazeinformation can include a focus of the eye with respect to a particulardisplay. The focus of the eye can be directed through a tunable opticalelement such as a tunable lens or mirror. Based on the gaze information,the tunable lens or mirror can be activated, or tuned, to refocus lightpathing through the lens or mirror. In some implementations, the tunableoptical elements can be located at a display. In this instance, thetunable optical elements can be activated or tuned at the display inreal time as the eye is tracked. The tunable optical elements caninclude one or more lenses, mirrors, or any combination thereof.

As such, the tunable optical elements can be adjusted based on themovement, or lack thereof, of the tracked eye. The tunable opticalelements can be used to provide dynamic focusing and defocusing in realtime. For example, the tunable optical elements can be used to solveinconsistencies between accommodation and vergence when viewing imagesat a VR or AR display.

FIG. 8 shows an example of a generic computer device 800 and a genericmobile computer device 850, which may be used with the techniquesdescribed here. Computing device 800 is intended to represent variousforms of digital computers, such as laptops, desktops, workstations,personal digital assistants, servers, blade servers, mainframes, andother appropriate computers. Computing device 850 is intended torepresent various forms of mobile devices, such as personal digitalassistants, cellular telephones, smartphones, and other similarcomputing devices. The components shown here, their connections andrelationships, and their functions, are meant to be exemplary only, andare not meant to limit implementations of the inventions describedand/or claimed in this document.

Computing device 800 includes a processor 802, memory 804, a storagedevice 806, a high-speed interface 808 connecting to memory 804 andhigh-speed expansion ports 810, and a low speed interface 812 connectingto low speed bus 814 and storage device 806. Each of the components 802,804, 806, 808, 810, and 812, are interconnected using various busses,and may be mounted on a common motherboard or in other manners asappropriate. The processor 802 may process instructions for executionwithin the computing device 800, including instructions stored in thememory 804 or on the storage device 806 to display graphical informationfor a GUI on an external input/output device, such as display 816coupled to high speed interface 808. In other implementations, multipleprocessors and/or multiple buses may be used, as appropriate, along withmultiple memories and types of memory. Also, multiple computing devices800 may be connected, with each device providing portions of thenecessary operations (e.g., as a server bank, a group of blade servers,or a multi-processor system).

The memory 804 stores information within the computing device 800. Inone implementation, the memory 804 is a volatile memory unit or units.In another implementation, the memory 804 is a non-volatile memory unitor units. The memory 804 may also be another form of computer-readablemedium, such as a magnetic or optical disk.

The storage device 806 is capable of providing mass storage for thecomputing device 800. In one implementation, the storage device 806 maybe or contain a computer-readable medium, such as a floppy disk device,a hard disk device, an optical disk device, or a tape device, a flashmemory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. A computer program product may be tangibly embodied inan information carrier. The computer program product may also containinstructions that, when executed, perform one or more methods, such asthose described above. The information carrier is a computer- ormachine-readable medium, such as the memory 804, the storage device 806,or a memory on processor 802.

The high speed controller 808 manages bandwidth-intensive operations forthe computing device 800, while the low speed controller 812 manageslower bandwidth-intensive operations. Such allocation of functions isexemplary only. In one implementation, the high-speed controller 808 iscoupled to memory 804, display 816 (e.g., through a graphics processoror accelerator), and to high-speed expansion ports 810, which may acceptvarious expansion cards (not shown). In the implementation, low-speedcontroller 812 is coupled to storage device 806 and low-speed expansionport 814. The low-speed expansion port, which may include variouscommunication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet)may be coupled to one or more input/output devices, such as a keyboard,a pointing device, a scanner, or a networking device such as a switch orrouter, e.g., through a network adapter.

The computing device 800 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 820, or multiple times in a group of such servers. Itmay also be implemented as part of a rack server system 824. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 822. Alternatively, components from computing device 800 may becombined with other components in a mobile device (not shown), such asdevice 850. Each of such devices may contain one or more of computingdevice 800, 850, and an entire system may be made up of multiplecomputing devices 800, 850 communicating with each other.

Computing device 850 includes a processor 852, memory 864, aninput/output device such as a display 854, a communication interface866, and a transceiver 868, among other components. The device 850 mayalso be provided with a storage device, such as a microdrive or otherdevice, to provide additional storage. Each of the components 850, 852,864, 854, 866, and 868, are interconnected using various buses, andseveral of the components may be mounted on a common motherboard or inother manners as appropriate.

The processor 852 may execute instructions within the computing device840, including instructions stored in the memory 864. The processor maybe implemented as a chipset of chips that include separate and multipleanalog and digital processors. The processor may provide, for example,for coordination of the other components of the device 850, such ascontrol of user interfaces, applications run by device 850, and wirelesscommunication by device 850.

Processor 852 may communicate with a user through control interface 848and display interface 856 coupled to a display 854. The display 854 maybe, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display)or an OLED (Organic Light Emitting Diode) display, or other appropriatedisplay technology. The display interface 856 may comprise appropriatecircuitry for driving the display 854 to present graphical and otherinformation to a user. The control interface 858 may receive commandsfrom a user and convert them for submission to the processor 852. Inaddition, an external interface 862 may be provide in communication withprocessor 852, so as to enable near area communication of device 850with other devices. External interface 862 may provide, for example, forwired communication in some implementations, or for wirelesscommunication in other implementations, and multiple interfaces may alsobe used.

The memory 864 stores information within the computing device 850. Thememory 864 may be implemented as one or more of a computer-readablemedium or media, a volatile memory unit or units, or a non-volatilememory unit or units. Expansion memory 854 may also be provided andconnected to device 850 through expansion interface 852, which mayinclude, for example, a SIMM (Single In Line Memory Module) cardinterface. Such expansion memory 854 may provide extra storage space fordevice 850, or may also store applications or other information fordevice 850. Specifically, expansion memory 854 may include instructionsto carry out or supplement the processes described above, and mayinclude secure information also. Thus, for example, expansion memory 854may be provide as a security module for device 850, and may beprogrammed with instructions that permit secure use of device 850. Inaddition, secure applications may be provided via the SIMM cards, alongwith additional information, such as placing identifying information onthe SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory,as discussed below. In one implementation, a computer program product istangibly embodied in an information carrier. The computer programproduct contains instructions that, when executed, perform one or moremethods, such as those described above. The information carrier is acomputer- or machine-readable medium, such as the memory 864, expansionmemory 854, memory on processor 852, or a propagated signal that may bereceived, for example, over transceiver 868 or external interface 862.

Device 850 may communicate wirelessly through communication interface866, which may include digital signal processing circuitry wherenecessary. Communication interface 866 may provide for communicationsunder various modes or protocols, such as GSM voice calls, SMS, EMS, orMMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others.Such communication may occur, for example, through radio-frequencytransceiver 868. In addition, short-range communication may occur, suchas using a Bluetooth, WiFi, or other such transceiver (not shown). Inaddition, GPS (Global Positioning System) receiver module 850 mayprovide additional navigation- and location-related wireless data todevice 850, which may be used as appropriate by applications running ondevice 850.

Device 850 may also communicate audibly using audio codec 860, which mayreceive spoken information from a user and convert it to usable digitalinformation. Audio codec 860 may likewise generate audible sound for auser, such as through a speaker, e.g., in a handset of device 850. Suchsound may include sound from voice telephone calls, may include recordedsound (e.g., voice messages, music files, etc.) and may also includesound generated by applications operating on device 850.

The computing device 850 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as acellular telephone 880. It may also be implemented as part of asmartphone 882, personal digital assistant, or other similar mobiledevice.

A number of applications can be implemented based on the conceptdescribed herein. For example, the TOF pixel as shown in FIGS. 1B and 1Ccan also be used to detect the user's facial characteristic, with theoption to include eye gesture tracking, for facial recognition oremotion detection. As another example, the eye gesture tracking based onthe implementations described herein can be used to replace orsupplement mouse to locate where the user's interest or focus at thedisplay. In some implementations, the eye gesture tracking based on theimplementations described herein can be used for more accurateadvertisement targeting or to predict user behavior. For example,machine learning with artificial neuro-network can be used to learn andrecognize the user behavior based on the where the user looks at thedisplay. Different weightings can be given for different contents basedon different user behaviors, such as (1) not looking (2) looking but donot choose (3) look and choose, where the weighting could be from smallto large. Furthermore, the duration of the user's focus on a particularcontent can also be used to record the user's interest level, forexample, the longer the duration, the higher the weighting. In someimplementations, displaying advertisement on a website can result indifferent charge to the payer of the advertisement based on the interestlevel received from the users, rather than the conventional click orno-click behaviors.

In some implementations, the eye gesture tracking based on theimplementations described herein can also be used for gaming. Forexample, referring to FIG. 2A, a car racing or aircraft flying game maybe played by a user on the mobile phone 224, or the tablet 230. Changesof user's eye gaze over time may be used to control a movement (e.g.,direction) of a car indicating where the user wants to go. As anotherexample, data collected by the accelerometer 211, the gyroscope 212, andthe eye-tracking module may be used to track movements of both theuser's head movement and eye gaze. In some implementations, a separatebutton may be included to control the speed and another optional buttonmay further be included to control an extra action such as shooting orswitching gear. The information of the head movement and the informationof the eye gaze, alone or combined, may be used by a game running on themobile phone or the tablet to determine an action of a user, and thegame may then respond accordingly. In this example, a combination of avector representing a turning position of the user's head and a vectorrepresenting a gaze of the user's eye may be used to determine the angleof the gaze with respect to the head, which may be used by the game tointerpret an action or a state of mind of the user.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. For example, various formsof the flows shown above may be used, with steps re-ordered, added, orremoved.

Embodiments of the invention and all of the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Embodiments ofthe invention can be implemented as one or more computer programproducts, e.g., one or more modules of computer program instructionsencoded on a computer readable medium for execution by, or to controlthe operation of, data processing apparatus. The computer readablemedium can be a machine-readable storage device, a machine-readablestorage substrate, a memory device, a composition of matter effecting amachine-readable propagated signal, or a combination of one or more ofthem. The term “data processing apparatus” encompasses all apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them. A propagated signal is an artificially generated signal, e.g.,a machine-generated electrical, optical, or electromagnetic signal thatis generated to encode information for transmission to suitable receiverapparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a standalone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a tablet computer, a mobile telephone, a personaldigital assistant (PDA), a mobile audio player, a Global PositioningSystem (GPS) receiver, to name just a few. Computer readable mediasuitable for storing computer program instructions and data include allforms of nonvolatile memory, media and memory devices, including by wayof example semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto optical disks; and CD ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, embodiments of the invention canbe implemented on a computer having a display device, e.g., a CRT(cathode ray tube) or LCD (liquid crystal display) monitor, fordisplaying information to the user and a keyboard and a pointing device,e.g., a mouse or a trackball, by which the user can provide input to thecomputer. Other kinds of devices can be used to provide for interactionwith a user as well; for example, feedback provided to the user can beany form of sensory feedback, e.g., visual feedback, auditory feedback,or tactile feedback; and input from the user can be received in anyform, including acoustic, speech, or tactile input.

Embodiments of the invention can be implemented in a computing systemthat includes a back end component, e.g., as a data server, or thatincludes a middleware component, e.g., an application server, or thatincludes a front end component, e.g., a client computer having agraphical user interface or a Web browser through which a user caninteract with an implementation of the invention, or any combination ofone or more such back end, middleware, or front end components. Thecomponents of the system can be interconnected by any form or medium ofdigital data communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the invention or of what may beclaimed, but rather as descriptions of features specific to particularembodiments of the invention. Certain features that are described inthis specification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

In each instance where an HTML file is mentioned, other file types orformats may be substituted. For instance, an HTML file may be replacedby an XML, JSON, plain text, or other types of files. Moreover, where atable or hash table is mentioned, other data structures (such asspreadsheets, relational databases, or structured files) may be used.

Particular embodiments of the invention have been described. Otherembodiments are within the scope of the following claims. For example,the steps recited in the claims can be performed in a different orderand still achieve desirable results.

What is claimed is:
 1. A system, comprising: a machine including adisplay, the display including a plurality of tunable optical elements;a device including circuitry configured to: obtain an electrical signalthat represents a measurement, by a photodetector, of an optical signalreflected from an eye, determine a depth map of the eye based on phasedifferences between a reference signal and the electrical signalgenerated by the photodetector, and determine a gaze information thatrepresents a gaze of the eye based on the depth map; and one or moreprocessors in communication with the machine and the device, the one ormore processors including one or more storage devices storinginstructions that are operable, when executed by the one or moreprocessors, to cause the one or more processors to perform theoperations including: receiving, from the device, output datarepresenting the gaze information; and determining the gaze informationrepresenting the gaze of the eye in relation to the display of themachine.
 2. The system of claim 1, wherein the operations furthercomprise: determining a particular position on the display that the eyeis focused on, the particular position being based on the gazeinformation representing the gaze of the eye in relation to the display;and providing an indication at the particular position on the display.3. The system of claim 1, wherein the operations further comprise:determining a particular position on the display that the eye is focusedon, the particular position being based on the gaze informationrepresenting the gaze of the eye in relation to the display; andproviding a foveated image at the particular area on the display.
 4. Thesystem of claim 1, wherein the plurality of tunable optical elementsinclude tunable lenses or tunable mirrors.
 5. The system of claim 4,wherein a tuning of a subset of the plurality of tunable opticalelements is activated based on the gaze information representing thegaze of the eye in relation to the display.
 6. The system of claim 5,wherein the tuning of the subset of the plurality of tunable opticalelements comprises dynamically refocusing light incident on the subsetof the plurality of tunable optical elements.
 7. The system of claim 1,further comprising: a wearable coupled to the machine, the device, andthe one or more processors to form an integrated hardware package, thedisplay of the machine being opaque in which visual images are shown onthe display by one or more of an array of light sources.
 8. The systemof claim 1, further comprising: a wearable coupled to the machine andthe device to form an integrated hardware package, the display of themachine being opaque in which visual images are shown on the display byone or more of an array of light sources; and the one or more processorslocated at a remote location and in communication with the integratedhardware package via a wireless or wired connection.
 9. The system ofclaim 1, further comprising: a wearable coupled to the machine, thedevice, and the one or more processors to form an integrated hardwarepackage, the display of the machine being at least partly transparent toimages projected towards the display, whereby a property of the imagesprojected towards the display is modified by one or more of theplurality of tunable optical elements of the display.
 10. The system ofclaim 1, further comprising: a wearable coupled to the machine and thedevice to form an integrated hardware package, the display of themachine being at least partly transparent to images projected towardsthe display, whereby a property of the images projected towards thedisplay is modified by one or more of the plurality of tunable opticalelements of the display; and the one or more processors located at aremote location and in communication with the integrated hardwarepackage via a wireless or wired connection.
 11. The system of claim 1,further comprising: a pluggable coupled to the device and the one ormore processors to form an integrated hardware package; and the machinelocated at a remote location and in communication with the integratedhardware package via a wireless or wired connection, the display of themachine being opaque in which visual images are shown on the display byone or more of an array of light sources.
 12. The system of claim 1,further comprising: a wearable couple to the device and the one or moreprocessors to form an integrated hardware package; and the machinelocated at a remote location and in communication with the integratedhardware package via a wireless or wired connection, the display of themachine being opaque in which visual image are shown on the display byone or more of an array of light sources.
 13. The system of claim 12,wherein the operations further comprise: determining a particularposition on the display that the eye is focused on, the particularposition being based on the gaze information representing the gaze ofthe eye in relation to the display; and providing an indication at theparticular position on the display.
 14. The system of claim 1, whereinthe optical signal reflected from the eye is generated by an opticalsource that is biased by a modulated signal, the modulation signal beingin sync with the reference signal.
 15. A device, comprising: a pluralityof tunable optical elements for adjusting focal lengths; one or moreprocessors in communication with the plurality of tunable opticalelements, the one or more processors including one or more storagedevices storing instructions that are operable, when executed by the oneor more processors, to cause the one or more processors to performoperations comprising: obtaining an electrical signal that represents ameasurement, by a photodetector, of an optical signal reflected from aneye, determining a depth map of the eye based on phase differencesbetween a reference signal and the electrical signal generated by thephotodetector, determining gaze information that represents a gaze ofthe eye based on the depth map, the gaze information representing thegaze of the eye in relation to a display of a remote device, andactivating a tuning of a subset of the plurality of tunable opticalelements based on the gaze information.